Method of displaying an image on a see-through display

ABSTRACT

A method of displaying an image on a see-through display comprises: obtaining a first electro-magnetic radiation matrix of radiation intensity values of an object; dividing the first matrix into a second matrix representing a first subset of the radiation intensity values, and a third matrix representing a second subset of the radiation intensity values; generating a first grayscale image with an enhanced contrast representing the first subset of the radiation intensity values from the second matrix; colouring the first grayscale image with a first colourmap to obtain a first colour image; generating a second grayscale image representing the second subset of the radiation intensity values; colouring the second grayscale image with a second colourmap to obtain a second colour image; combining the first colour image and the second colour image; and displaying the combined colour image on the see-through display.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 16/767,669 filed on May 28, 2020, which was a national stageapplication of PCT International Application No. PCT/EP2017/083934 filedon Dec. 20, 2017, the entire disclosures of which are incorporatedherein by reference.

TECHNICAL FIELD

The present invention relates to a method of displaying an image, suchas a thermal image, on a see-through display. More specifically, thedisplayed image would otherwise be non-visible for the user of thesee-through display. The invention also relates to a correspondingimaging system and to a computer program product for implementing themethod.

BACKGROUND OF THE INVENTION

In various fields, it would be useful to show non-visible information,such as thermal information, on a transparent or see-through display,referred to also as an augmented reality display, for a user. This couldbe particularly useful for example for firefighters, who often encounterdifficulties to see through thick smoke. Currently existing hands-freethermal vision systems rarely use superior see-through displays asdisplaying thermal images on such displays while respecting the way theuser perceives them is badly understood. Currently commerciallyavailable products can be divided into handheld thermal cameras used forfirefighting for example, hands-free thermal vision devices used forfirefighting for example, and augmented vision devices used in otherfields of applications.

Handheld firefighting thermal cameras use liquid crystal display (LCD)screens to provide a “live” thermal image to the firefighter. Dependingon the camera model, the associated thermal image processing ranges fromvery simple (black and white images with limited image enhancement) tomore complex (using multiple image enhancement techniques for increasingcontours and details of objects) with multiple colour schemes. However,the image processing and optimisation carried out for standard LCDscreens cannot often be used in the context of see-through displays (forexample because black and white thermal images are very faintlyperceived). As far as hands-free thermal vision devices are concerned,only few commercially available devices exist. These devices aretypically based on LCD screens, displayed in a glance mode (i.e. out ofcentral vision). Augmented vision devices for other fields ofapplications may be used for instance in military (e.g. pilot helmets),medical (augmented reality assisted surgery) and driving (head-updisplays) applications and they use similar concepts for displayinginformation in a partially nonobtrusive manner. However, especially whencompared to the needs of thermal imaging or firefighting, therequirements for the image processing are quite different.

An ideal augmented vision system displays non-visible information insuch a manner that it only adds information to the already visibleinformation (this is how seamlessness of the system is defined) asopposed to a system which would present a high level of obtrusiveness,preventing the user from accessing important visible information. Thisgoal is similar to various sensor fusion applications, where two (ormore) images from different modalities are mixed together in order tomaximise the resulting information. However, there are some importantdistinctions between traditional sensor fusion applications and imagingapplications for see-through displays. Firstly, in sensor fusionapplications, the user has an unmitigated control over the final image,which is not the case with transparent see-through display applications,where it is only possible to superpose onto the final image as perceivedby the user. Secondly, the dynamic range of real world lightingapplications is far greater than that of the augmented reality displays,which poses the problem of how to show relevant information in alllighting situations. Thirdly, traditional sensor fusion applicationshave mostly focused on how to blend images in order to maximise detailperception. However, for example in the firefighting domain, both thedetail perception and the temperature perception (understanding theexact temperature of an object) are important.

Thermal image processing has been studied for a wide variety ofapplications. However, in most if not in all of the cases, theinformation value has come from either the structure (thermal shapes) orthe metric value (temperatures). However, in some fields, such asapplications for firefighters, both the structure and metric value areof importance, because firefighters use a thermal camera for dangeroussituation assessment. This leads to two major problems: how to compressthe thermal image to maximise detail perception while maintaining goodtemperature perception, and how to colourise the resulting image. Mostof the currently known image compression techniques to compress anincoming thermal image to a more reduced range image rely on finding anoptimal histogram equalisation technique. However, these techniques aretypically applicable to static images only. Furthermore, existingsolutions to colourise a thermal image are not suited to firefightingapplications, for example. The existing solutions mostly focus oncolourising images with natural daytime appearance. Other colour schemesare usually limited to two types: single colour schemes (e.g. black tored colourmaps) and rainbow schemes (high number of colours). The needsfor firefighters, for example, are however not covered by thesetechniques.

SUMMARY OF THE INVENTION

It is an object of the present invention to overcome at least some ofthe problems identified above related to displaying electromagneticradiation information on a see-through display.

According to a first aspect of the invention, there is provided a methodof displaying an image on a see-through display.

-   -   The proposed new solution has the following advantages: Good        perception of contours of objects and physical elements (such as        walls, floor, furniture) to enhance spatial orientation.    -   Good perception of temperature of objects (if temperature is of        interest) based on an estimate of a level of danger.    -   Robustness of the displayed image towards environmental        conditions, such as lighting conditions, scene information etc.    -   Unobtrusiveness of the displayed image towards the perception of        the real world. Possible visual cues are visible at all times        and are not blocked by the displayed image.

According to another aspect of the invention, there is provided animaging system for displaying an image on a see-through display.

Other aspects of the invention are recited in the claims attachedhereto.

BRIEF DESCRIPTION OF THE DRAWINGS

Other features and advantages of the invention will become apparent fromthe following description of a non-limiting example embodiment, withreference to the appended drawings, in which:

FIG. 1 shows schematically some hardware components, which may be usedto implement the proposed method according to an example of the presentinvention;

FIG. 2 shows an example grayscale image obtained by a thermal sensoraccording to an example of the present invention;

FIG. 3 shows a histogram of the image of FIG. 2 according to an exampleof the present invention;

FIG. 4 shows a histogram of a lower temperature image part of FIG. 2according to an example of the present invention;

FIG. 5 shows an equalised histogram of the histogram of FIG. 4 accordingto an example of the present invention;

FIG. 6 shows a contrast enhanced lower temperature grayscale image partfor the image of FIG. 2 according to an example of the presentinvention;

FIG. 7 shows a rescaled higher temperature grayscale image part of theimage of FIG. 2 according to an example of the present invention;

FIG. 8 shows a histogram of the image of FIG. 7 according to an exampleof the present invention;

FIG. 9 shows a colourised lower temperature image part obtained from theimage of FIG. 6;

FIG. 10 shows a colourised higher temperature image part obtained fromthe image of FIG. 7;

FIG. 11 shows a nested colourmap used to colourise the lower and highertemperature image parts of FIGS. 6 and 7, respectively, according to anexample of the present invention;

FIG. 12 shows an alpha mask obtained from the higher temperature imagepart of FIG. 7 according to an example of the present invention;

FIG. 13 shows a final colourised blended image obtained from the imagesof FIGS. 9 and 10; and

FIG. 14 is a flow chart illustrating the proposed method according to anexample of the present invention.

DETAILED DESCRIPTION OF AN EMBODIMENT OF THE INVENTION

An embodiment of the present invention will now be described in detailwith reference to the attached figures. This embodiment is described inthe context of a firefighting application, but the teachings of theinvention are not limited to this environment. For instance, theteachings of the present invention could be used in any other scenario,where thermal information would add information, such as securityapplications, heavy industry (metallurgy, cement works) applications,specific sports, medical applications etc. Also, the teachings of thepresent invention are also not specifically tied to thermal imaging, butthey could be adapted to other sensors, such as ultraviolet or radarsensors, to show non-visible information in a seamless manner. Identicalor corresponding functional and structural elements which appear in thedifferent drawings are assigned the same reference numerals.

The present invention is in the field of augmented vision, a term whichmay be defined as the enhancement of the human visual system bypresentation of non-visible (yet physical) information by usingtransparent field of view or vision displays, also referred to asaugmented or mixed reality (AR/MR) displays. More specifically, theteachings of the present invention are particularly useful in thecontext of critical and emergency applications, where a quickunderstanding of information is crucial. The non-visible informationconsidered may be electromagnetic radiation in the infrared spectralrange. It typically extends from the nominal red edge of the visiblespectrum at 700 nanometres (frequency 430 THz) to 1 millimetre (300GHz). Thus, the electromagnetic radiation may be thermal radiation andemitted by an object enshrouded in smoke and for this reason normallynot visible. However, the teachings of the present invention are alsoapplicable to electromagnetic radiation in other spectral ranges.

The present invention is based on an algorithm, which processes thermalimages or electromagnetic radiation images more broadly in order todisplay them on a see-through display in the best possible way. The“seamlessness” of the displayed image depends on how the non-visibleinformation has been processed to maximise understanding of the mixed(visible+non-visible) image, how the image has been adapted to the useof a transparent display, and how it has been adjusted or calibrated tothe current environment. The present invention defines models,algorithms and/or testing procedures needed to achieve the userperception of “seamlessness”. The two major parts of this algorithm orprocess are briefly explained next.

A balance between details and thermal perception through a nestedcolourmap: The present invention uses two different specificallydesigned colourmaps to achieve two separate goals. This is believed tobe the optimal way of displaying a thermal image with the goal ofmaximising both detail and temperature perception. This approach couldbe used on normal displays as well. A colourmap may be defined as alook-up table for matching input grayscale values to colour values.Prior to applying the colourmaps, a specific histogram equalisationtechnique is used as explained later in more detail. Histogramequalisation is a technique used for adjusting image values to enhancecontrast.

Specific adaptation to transparent displays: Due to the presentation ofan image directly in the field of view of the user, AR displays tend tomaximise the defects of the image stream, and can rapidly becomeuncomfortable to wear if no extra care has been taken to minimise thesedefects. The techniques proposed for brightness or luminance adaptation(also display transparency adaptation) tackle the largest perceptualproblems of any augmented vision system.

FIG. 1 schematically illustrates the hardware components which may beuseful for understanding the teachings of the present invention. Ahelmet 1, in this example a firefighting helmet, is designed to be wornby a firefighter. A thermal camera component or unit 3 is installed atthe front part of the helmet and in this example comprises a thermalcamera or sensor 5 and a luminosity sensor 7. The thermal camera 5 isconfigured to capture one or more electromagnetic radiation frames, inthis example thermal image frames or simply thermal frames, of theenvironment. A thermal frame is understood to be a matrix oftemperatures as detected or measured by the thermal camera. A thermalframe may then be visualised as a thermal image so that in this examplefor each image pixel there is a corresponding temperature matrix valuein the temperature matrix. The temperature values of the temperaturematrix can thus be simply converted into encoded image pixel values.When multiple frames are taken, then these frames may be shown as avideo for the user. In this example, each of the matrix element valuesis encoded in 14 bits. For this reason, the thermal frame may be calleda 14-bit temperature matrix. A modified thermal image as will beexplained later may be shown on a display 9, which in this example is asee-through display 9. A see-through display is an electronic display,which allows the user to see what is shown on the (glass) screen whilestill being able to see through it. The see-through display 9 may havean integrated display brightness control unit or this unit may beprovided separately. In FIG. 1, there is also shown a breathing mask 11for the firefighter. It is to be noted that instead of being mounted onthe helmet 1, the thermal camera 5 and/or the luminosity sensor 7 couldbe mounted on the breathing mask 11 or somewhere else. A data processingor software unit, which is not shown in FIG. 1, is configured to processthe thermal frames prior to displaying the modified or processed thermalframes on the see-through display 9. A wireless or wired datacommunication link is provided between at least some of the followingcomponents: the thermal camera component 3, the data processing unit,the brightness control unit and the see-through display 9.

As mentioned earlier, both the details and the temperature perception(understanding the exact temperature of an object) are important forfirefighting applications. However, in data visualisation, these areopposing goals, namely quantity reading/identification task(temperature) and form perception (details). To arrive at the presentinvention, findings of the data visualisation were first validated bycarrying out psycho-perceptual experiments in which the observers weregiven two separate tasks: compare pairs of images in terms of number ofdetails, and estimate the value of a portion of a displayed image. Eachof these tasks were repeated multiple times using different colourschemes representing the various possibilities offered by datavisualisation. These experiments were performed on a normal computerscreen by blending a thermal image and a visual image together tosimulate the effect of using a transparent system, and by using aspecific AR display model. It was quickly concluded that one “ideal”colourmap was not possible, as multi-colour colourmaps gave betterresults on the temperature estimation task, while single colourcolourmaps worked better on the detail perception as will be explainedbelow in more detail.

According to one example of the present invention, a system and a methodare provided for processing and displaying thermal images on asee-through display for firefighting applications. The system is thusconfigured to carry out the method. The processing of the originalthermal frame is in this example divided into three phases as summarisedbelow and explained later in more detail:

-   -   1. Automatic gain control: The original thermal frame (input        frame or matrix for the processing unit), which can be        visualised as an original thermal image as shown in FIG. 2 and        captured by the thermal camera 5, is processed in order to lower        the input dynamic range to the display output dynamic range.        This involves dividing the first temperature matrix, also        referred to as the input temperature matrix, into two matrices        of the same size: a second or lower temperature matrix        containing all the temperatures below a specific threshold and a        third or higher temperature matrix containing all the        temperatures above or equal to this threshold. The lower        temperature matrix is then non-linearly transformed into an        image of a lower dynamic range (a lower temperature image),        while the higher temperature matrix is linearly transformed into        an image of a lower dynamic range (a higher temperature image).        By automatic gain control is thus understood a process through        which the dynamic range of the input thermal frame or        temperature matrix is reduced towards the display dynamic range        while maintaining good contrast. Dynamic range may be defined as        the ratio of an input or output maximum value to minimum value.        The dynamic range of a thermal camera is typically higher than        the dynamic range of a display.    -   2. Colourisation: The lower temperature image is then colourised        by using a first colourmap, referred to also as a lower        temperature colourmap, while the higher temperature image is        then colourised by using a second colourmap, referred to also as        a higher temperature colourmap, which in this example is        different from the first colourmap (although they could be        substantially the same colourmap). These two colourmaps have        been designed to achieve separate goals: for the lower        temperature image to maximise form perception; and for the        higher temperature image to maximise metric data value        estimation. The two images are then blended or mixed into one        single continuous (in terms of colours) image, thanks to the        nested properties of the colourmaps.□    -   3. Automatic brightness control: The colourised mixed image is        then displayed on the see-through display 9. For this purpose,        the display brightness may be adapted, in this example based on        two factors: the estimated information value of the scene (i.e.        the original thermal frame), and the current ambient or        background light level. Low information scenes lead to a lower        display or screen brightness (more transparent perceived image),        while maintaining a specific luminosity contrast between the        displayed image and the background scene. The automatic        brightness control is thus a process through which the display        backlight drive or more specifically its value is computed based        on the scene's informational value and/or the ambient light        level obtained from the luminosity sensor 7.

The automatic gain control process is next explained in more detail. Theprocess uses a new global histogram equalisation technique (global inthe sense that the technique is applied to the whole thermal frame to beprocessed), which aims to satisfy the two separate goals of thermalimage perception (details and temperature). This is achieved bythresholding the input temperature matrix into two separate matriceswith the lower temperature matrix representing the lower temperatures,and the higher temperature matrix representing the higher temperatures.FIG. 2 shows the visualisation of the original temperature matrix as theoriginal thermal image while FIG. 3 illustrates the original histogramfor that image. The peak at the right end of the histogram is caused bythe thermal camera saturation. FIG. 3 also shows the temperaturethreshold, which in this example is set to 80° C. However, other (fixed)temperature threshold values are equally possible. The temperaturethreshold may be between 40° C. and 120° C., or between 60° C. and 100°C. or more specifically between 70° C. and 90° C. It is to be noted thatthat the thermal image of FIG. 2 and the histogram of FIG. 3 are shownmerely for illustrative purposes but the proposed method does not infact use the thermal image of FIG. 2 or the histogram of FIG. 3 in thecomputations. Each of the lower and higher temperature matrices is thentreated in a different manner. The lower temperature matrix isnon-linearly compressed or dilated to increase contrast by using anadapted version of a standard histogram equalisation technique withboundaries put on the compression (or dilation) factor as seen inAlgorithm 1 given below. The reason for using this specific histogramequalisation technique is to increase contrast while limiting the numberof visual artefacts resulting from a classical histogram equalisationtechnique (i.e. partial linearity can be maintained by the proposedmethod). In other words, the resulting histogram is not completely flat,but only approximately flat as shown in FIG. 5. It is to be noted thatthe histogram shown in FIG. 5 shows fewer than 256 histogram bins and isthus a simplified version of the real histogram. The non-flatness thusmeans that visual image artefacts can be minimised. In this manner, thelower temperature image part is contrast enhanced through this specificnon-linear histogram equalisation technique and mapped to the [0, 255]encoded image element value range. It is to be noted that a histogramequalisation process is by nature a non-linear process.

The developed histogram equalisation technique used to process the lowertemperature matrix functions as follows:

-   -   1. All pixels (or image elements more broadly) having a value        higher than the temperature threshold are ignored in the future        calculations.    -   2. The total number of pixels is divided by the target histogram        bins (256 in this example). This gives the target pixel count        per histogram bin bin_(limit). If all the histogram bins contain        the same number of pixels, the target histogram is completely        flat and thus perfectly equalised. However, in this example, the        proposed method does not lead to a perfectly equalised        histogram.    -   3. A histogram as shown in FIG. 4 is obtained for the lower        temperature matrix, which is referred to as an input histogram,        which can be defined as a vector of number of pixels for each        temperature value such that each temperature value of the lower        temperature matrix defines an input histogram bin.    -   4. Each bin of the input histogram is considered, and a new        histogram, referred to as an output histogram, is obtained by        using a pseudo code described in Algorithm 1 as shown below. It        is to be noted that the algorithm considers pixels in one single        input histogram bin as a single entity, i.e. they are all        allocated to one output histogram bin. Each value b_(input) of        the input histogram (i.e. the number of pixels in a particular        input histogram bin) is added to the current bin of the output        histogram, b_(output)[ind_(output)] indicating the number of        pixels in a particular output histogram bin. As long as the        number of pixels in the current bin of the output histogram has        not reached or surpassed bin_(limit) (first condition), the        output index ind_(output) does not change, i.e. the process        keeps adding pixels from input bins (bins by bins) to the        current bin of the output histogram. It is also verified that        the current output histogram bin does not span over a too large        range of input histogram bins by comparing the difference of the        current input bin index ind_(input) and the last index        ind_(inputlast), where the process switched to a “new” output        bin, with the compression limit compression_(limit) (second        condition). If the difference exceeds the compression_(limit)        (expressed as a number of bins), the output bin index is        incremented, i.e. the process switches to filling the next        output histogram bin. In other words, the process keeps adding        pixels to the current bin of the output histogram until        whichever of the first and second conditions is fulfilled. Then        the process starts filing the next output histogram bin. The        compression_(limit) may be between 5 and 100, or 5 and 50 or        more specifically between 5 and 20 bins. It is to be noted that        the second condition is the novel feature of the present        histogram equalisation technique. The output histogram obtained        in this manner is thus an equalised version of the input        histogram.    -   5. If the new histogram contains more than 256 bins, the result        is linearly remapped to 256 bins (or any other given number of        bins). If the output histogram contains at most 256 bins, the        histogram is not remapped.    -   6. A histogram back projection is calculated by remapping each        pixel of the lower temperature matrix to the intended value in        the [0, 255] range by using the new histogram. This may be done        for example by starting from one temperature extreme (e.g. the        lowest temperature) of the lower temperature matrix and        allocating the encoded value of the first bin of the equalised        histogram to the lowest temperature values. If there are still        some pixels left in the first bin of the equalised histogram,        then the process moves to the second lowest temperature values        and allocates the first bin encoded value also to the second        lowest temperatures. Once there are no more pixels left in the        first bin, the process moves to the second bin and assigns the        encoded value of this bin to next available temperature values        in the lower temperature matrix. In this manner, all the        temperature values of the lower temperature matrix are allocated        encoded pixel values in order of increasing temperature values        of the matrix. Thus, the back projection of the equalised        histogram may be considered to be a re-application of the        equalised histogram to the lower temperature matrix functioning        as a look-up table for pixel brightness values.    -   7. This gives the contrast enhanced lower temperature image part        as shown in FIG. 6.

Algorithm 1: Custom histogram equalisation technique   ind_(input) = 0ind_(output) = 0 ind_(inputlast) = 0  for all b_(input) do  b_(output)[ind_(output)] = b_(output)[ind_(output)] +b_(input)[ind_(input)]   if b_(output)[ind_(output)] ≥ bin_(limit) then  ind_(output) = ind_(output) + 1   ind_(inputlast) = ind_(input)  elseif (ind_(input) − ind_(inputlast)) ≥ compression_(limit) then  ind_(output) = ind_(output) + 1   ind_(inputlast) = ind_(input)  endif  ind_(input) = ind_(input) + 1 end for

As far as the higher temperature matrix is concerned, it is simplylinearly scaled or mapped to match the limited range of 256 encodedimage element values (or any other given number of encoded values). Thefollowing equation defines the linear mapping equation for the highertemperature matrix/imagepix_(output)=255×(temp_(input)−temp_(threshold)/temp_(max)−temp_(threshold)).

Each pixel value pix_(output) or image element value of the rescaledtemperature matrix is thus calculated by using the above equation. Eachpixel pix_(output) is calculated based on the corresponding inputtemperature temp_(input) at the same location in the higher temperaturematrix. In the above equation, temp_(threshold) is the temperaturethreshold (80° C. in this case) and temp_(max) is the maximumtemperature of the thermal camera 5. The division operation gives avalue between 0 and 1, and by multiplying it by 255, the desired rangeis achieved. The resulting modified or processed higher temperatureimage part and its histogram are shown in FIGS. 7 and 8, respectively.It is to be noted that, here again, the histogram of FIG. 8 of the imageof FIG. 7 is merely shown for illustrative purposes, but the generationof this histogram is optional and it is not used in the proposed method.Furthermore, the histogram shown in FIG. 8 shows fewer than 256histogram bins and is thus a simplified version of the real histogram.

The colourisation process is explained next in more detail. In thisprocess, the processed lower temperature and higher temperature imageparts, which are in this example 8-bit grayscale, black-and-white ormonochrome images (i.e. each pixel is encoded in 8 bits), are taken anda colour image, which in this example is a 24-bit image (i.e. each pixelis encoded in 24 bits) is generated. This process of colourisingotherwise black-and-white univariate information is calledpseudocolouring. Data visualisation theory defines two kinds of piecesof information included in images: metric (or value) which denotes thequantity stored at each point, and form which denotes the shape andstructure of the surface.

As mentioned earlier, the first colourmap is used to maximise formperception (details and contours of the scene). In order to do this, thefirst colourmap is selected as a single colour colourmap comprisingvalues of one colour. The first colourmap is a sequence of colourvalues, which vary monotonically in lightness and chromaticity. Incolour theory, lightness can be considered a representation of variationin the perception of a colour or colour space's brightness. It has adirect relation with relative luminance (same definition as for theluminance but bound to values [0,100]). Chromaticity is the definitionof what “colour” a specific pixel or image element is perceived,regardless of its luminance. The first colourmap can be visually shownas a line comprising a given number of connected colour points (in thisexample 256) each having a different colour value. In this example, thelightness or brightness of the colours in the first colourmap becomebrighter when moving towards the right end of the first colourmap. Inthe present example, the colour chosen for the first colourmap is blue,but any other suitable colour could be chosen instead. The firstcolourmap in this example thus comprises 256 different values of bluefor colourising the processed lower temperature image. It is to be notedthat in this example, each colour value in the first and secondcolourmaps is defined by thee colour channel components each definedwith 8 bits. The processed lower temperature grayscale image is thencolourised with the first colourmap to obtain a colourised and processedlower temperature image. A grayscale version of that image is shown inFIG. 9.

The second colourmap is used to maximise metric data value estimation,i.e. the capacity of the user to estimate the value (here temperature)of a specific part of the image. This is implemented by maximising thenumber of perceptually distinct colour sectors (just-noticeabledifference (JND)) in the second colourmap but with all colours sharingsimilar equal visual importance. It is estimated that in firefightingapplications, a ±10° C. approximation is acceptable in a temperaturerange between 80° C. and 680° C. It corresponds to 60 separate coloursectors. Also the second colourmap can be visually represented by a linecomprising a given number of connected colour points (in this example256) each having a different colour value. The second colourmap is inthis example built around 4 distinct main colours and interpolatedlinearly between these colours, selected in such a way to achieveJNDs >60. These main colours from left to right are in this examplewhite, yellow, orange and red. A grayscale version of a colourised andprocessed higher temperature image is shown in FIG. 10. This image isobtained by colourising the processed higher temperature grayscale imagewith the second colourmap.

The first and second colourmaps can be combined to obtain a nested orcombined colourmap consisting of the first and second colourmaps asshown in FIG. 11, the first colourmap being the left half of the nestedcolourmap, while the right half is the second colourmap. In thisexample, the first and second colourmaps are connected in such a mannerthat the connecting colour values of the first and second colourmapshave substantially the same chromaticity and lightness values. It is tobe noted that it is not necessary to physically connect or combine thefirst and second colourmaps, but preferably a colour value at one end ofthe first colourmap has lightness and chromaticity values which are thesame as the ones of a colour value at one end of the second colourmap toprovide a seamless link between the two colourmaps and thus to avoidartefacts in the image. The first and second colourmaps can be said tobe static in the sense that they remain constant for multiple thermalframes, for example for the entire duration of one or more videosconsisting of a set of consecutive image frames.

The two colour images are then combined or blended using an alpha maskshown in FIG. 12. The alpha mask, which is a binary image or matrix ofthe same size as the original temperature matrix, is derived from theoriginal thermal frame so that the temperature values higher than thethreshold temperature are given a first value, while the temperaturevalues smaller than or equal to the threshold temperature are a second,different value. More specifically, the image element or pixel values ofthe alpha mask are either 0 or 1. In this example, pixel values of thealpha mask are 1 wherever the temperature values in the original thermalframe are above the temperature threshold, which in this example is 80°C. Other pixel values in the alpha mask are set to 0. The alpha maskindicates how the colourised and processed higher temperature imageshould be superimposed on the colourised and processed lower temperatureimage. In other words, the values 1 in the alpha mask indicate the pixellocations where the colourised and processed higher temperature imageshould replace the pixels of the colourised and processed lowertemperature image. Instead of replacing pixels, the blended image may beobtained as a completely new image starting from the colourised lowerand higher temperature images. FIG. 13 shows a grayscale version of thefinal blended colour image.

The automatic brightness or luminosity control process is next explainedin more detail. The luminosity of the display and its correspondingluminance is adapted to the luminance of the background such that boththe visible background and thermal overlay information areunderstandable. Luminosity is defined as the total light emitted by thefull display module, and more specifically the total light emitted bythe backlight drive. On the other hand, luminance is defined by how muchluminous energy is detected by a human eye when looking at a surface(either the background or the display) at a given angle of view. Itdefines how bright the surface looks. The display and the backgroundneed to keep a fixed luminance ratio if it is desired that the screenalways appears “equally” bright. The luminosity or luminance adaptationis implemented by using an integrated or separate backlight in thedisplay 9 and the forward-looking luminosity sensor 7. In order to findthe right parameters for their relation, both the display 9 andluminosity sensor 7 are first characterised.

-   -   For the display 9, a spectroradiometer is used at various        backlight intensities. The goal is to measure the overall        display transmissivity, the luminance values of all individual        display colours at a fixed backlight level as well as the        varying luminance for all possible backlight levels.    -   The luminosity sensor 7 is either pre-calibrated, or if needed,        the characterisation is carried out by using a trusted light        source, along with colour filters with known translucent        properties. In this manner, the response of the sensor to        different colours at different light levels can be established.

In addition to the goal of maintaining a correct ratio of displayluminance to scene luminance, the automatic brightness control isoptionally also responsible for adapting the luminance of the displaydepending on the scene's (image's) information value. This value may bedetermined by the total dynamic range of the original thermal frame. Alow dynamic range typically implies a final thermal image with lowinformation value, e.g. when the user is looking directly at a wallhaving only a very limited temperature range. In these cases, theluminance (or brightness) of the display is adapted in such a way thatthe display or the displayed image is seen as more transparent. Thescene information value is computed to stay within [0:1] range.

If both the scene luminance and the scene information value areconsidered, then the automatic brightness control is limited by fourseparate thresholds:

-   -   A lower absolute threshold backlight_(low) under which the        display backlight drive value is not diminished;    -   An upper absolute threshold backlight_(high) over which the        display backlight drive value is not increased;    -   a lower ratio threshold ratio_(low), which is a fixed ratio of        the display luminance to the scene luminance, and corresponds to        low scene information value, which is a value typically slightly        higher than 0. The lower ratio threshold ratio_(low) may be        chosen empirically and may be a value between 1 and 1.4 or more        specifically a value between 1.1 and 1.3, such as 1.2; and    -   an upper ratio threshold ratio_(high), which is a fixed ratio of        the display luminance to the scene luminance, and corresponds to        normal scene information value, which is a value typically equal        to or slightly below 1. The upper luminosity ratio threshold        ratio_(high) may be chosen empirically and may be a value        between 1.8 and 2.2 or more specifically a value between 1.9 and        2.1, such as 2.

The full automatic brightness control algorithm according to one exampleis described in Algorithm 2 below. The target luminance ratiolum_(ratio) (the display luminance divided by the scene luminance) isfirst calculated by multiplying the scene_(information) value with theupper ratio threshold ratio_(high). It is then determined whether or notthe obtained value is under the lower ratio threshold ratio_(low), andif it is, then the lum_(ratio) is set it to this threshold value. Thescreen luminance lum_(screen) is then calculated by multiplying thelum_(ratio) with the measured scene luminance lum_(scene). Now thescreen luminance is compared with the two absolute thresholdsbacklight_(low) and backlight_(high), and set it to one of theseboundary values if the screen luminance would otherwise be lower thanbacklight_(low) or higher than backlight_(high). According to thisexample, the lum_(ratio) varies depending on the scene informationvalue. In this example, scene information values between the lower andupper thresholds result in linearly increasing display backlight drivevalues.□

Algorithm 2: Automatic brightness control technique   lum_(ratio) =scene_(information) × ratio_(high) if lum_(ratio) ≤ ratio_(low) then lum_(ratio) = ratio_(low) end if lum_(screen) = lum_(ratio) ×lum_(scene) if lum_(screen) ≤ backlight_(low) then  lum_(screen) =backlight_(low) else if lum_(screen) ≥ backlight_(high) then lum_(screen) = backlight_(high) end if

The flow chart of FIG. 14 summarises the proposed method of displayingnon-visible information on the see-through display 9. In step 101, theoriginal thermal frame is obtained by using the thermal camera 5. Inother words, the original thermal frame is made available by the thermalcamera 5 as a temperature matrix with a high dynamic range DR_(H). Instep 103, three matrices of the same size and shape as the originaltemperature matrix are generated from the original temperature matrixobtained in step 101:

-   -   The lower temperature matrix TM_(L) comprises all the        temperature values below or equal to the temperature threshold,        the other temperature values are set to 0;    -   The higher temperature matrix TM_(H) comprises all the        temperature values above the temperature threshold, the other        temperature values are set to 0;    -   The alpha mask, map or matrix TM_(A) whose matrix values are set        to 1 for all the non-zero values of TM_(H) and 0 for the other        matrix element values.

In step 105, the histogram, referred to as the input histogram, for thelower temperature matrix is generated. In step 107, the input histogramis equalised as explained above to obtain the equalised outputhistogram. In step 109, the contrast enhanced lower temperaturegrayscale image is generated from the equalised histogram and from thelower temperature matrix TM_(L). Thus, in steps 105, 107 and 109, thelower temperature matrix TM_(L) is non-linearly mapped to the lowertemperature grayscale image with a short dynamic range DR_(S) by usingthe histogram equalisation technique. This process also leads toobtaining a modified lower temperature matrix so that the lowertemperature image can be derived from the modified lower temperaturematrix. In step 111, the lower temperature grayscale image is colourisedby using the first colourmap to obtain the lower temperature colourimage C_(L).

In step 113, the higher temperature matrix TM_(H) is linearly mapped tothe higher temperature grayscale image with a short dynamic rangeDR_(S). This involves obtaining a modified higher temperature matrix sothat the higher temperature grayscale image can be derived from themodified higher temperature matrix. In step 115, the higher temperaturegrayscale image is colourised by using the second colourmap to obtainthe higher temperature colour image C_(H).

In step 117, the colour images C_(H) and C_(L) are blended using thealpha map TM_(A) to obtain the combined colour image C_(F) with thefollowing formula C_(F)=C_(L) TM_(A)*C_(H). In step 119, the combinedcolour image C_(F) is transmitted either wirelessly or through a cableto the display 9. In step 121, the value of the display backlight driveis determined based on the scene's information value derived from theoriginal input thermal frame and/or luminosity sensor input value. Instep 123, the combined colour image C_(F) is displayed on thesee-through display 9 with the display backlight drive set to the valuedetermined in step 121.

While the invention has been illustrated and described in detail in thedrawings and foregoing description, such illustration and descriptionare to be considered illustrative or exemplary and not restrictive, theinvention being not limited to the disclosed embodiment. Otherembodiments and variants are understood, and can be achieved by thoseskilled in the art when carrying out the claimed invention, based on astudy of the drawings, the disclosure and the appended claims. Forexample, instead of using the histogram equalisation technique asexplained above, any other process of enhancing contrast could be usedto process the lower temperature image part. Thus, any suitable standardhistogram equalisation technique could be used instead of the techniquedescribed above.

In the claims, the word “comprising” does not exclude other elements orsteps, and the indefinite article “a” or “an” does not exclude aplurality. The mere fact that different features are recited in mutuallydifferent dependent claims does not indicate that a combination of thesefeatures cannot be advantageously used.

The invention claimed is:
 1. A method of displaying an image on asee-through display, the method comprising: obtaining a first matrix ofelectromagnetic radiation of an object, the first matrix comprisingfirst matrix elements representing radiation intensity values ofcorresponding locations of the object; dividing the first matrix into asecond matrix representing a first subset of the radiation intensityvalues below or equal to a predefined threshold, and a third matrixrepresenting a second subset of the radiation intensity values above thepredefined threshold; generating a first grayscale image with anenhanced contrast representing the first subset of the radiationintensity values from the second matrix; colouring the first grayscaleimage with a first colourmap to obtain a first colour image; generatinga second grayscale image representing the second subset of the radiationintensity values; colouring the second grayscale image with a secondcolourmap, which is different from the first colourmap, to obtain asecond colour image; combining the first colour image and the secondcolour image to obtain a combined colour image of the same size andshape as the first colour image or the second colour image; anddisplaying the combined colour image on the see-through display.
 2. Themethod according to claim 1, wherein the step of generating the firstgrayscale image is achieved via a histogram equalization technique. 3.The method according to claim 1, wherein the step of generating thefirst grayscale image of the second matrix includes the sub-steps of:generating a first histogram for the second matrix; equalising the firsthistogram to obtain an equalised second histogram; and using theequalised second histogram to generate the first grayscale image.
 4. Themethod according to claim 3, wherein the first histogram comprises firsthistogram bins, while the second histogram comprises second histogrambins, and wherein the step of equalising the first histogram comprisesallocating image elements of the first histogram bins to the secondhistogram bins, such that all the image elements of a single firsthistogram bin are allocated to one second histogram bin, and such thatfilling a particular second histogram bin with the image elements of thefirst histogram bins is stopped when at least one of the followingevents happens: a number of image elements in the particular secondhistogram bin reaches a threshold value; and a number of different firsthistogram bins used to fill the particular second histogram bin reachesanother threshold value.
 5. The method according to claim 1, wherein thestep of generating the second grayscale image representing the secondsubset of the radiation intensity values is achieved by linearly mappingthe second subset of the radiation intensity values to a given number ofencoded radiation intensity values.
 6. The method according to claim 5,wherein the first colourmap and the second colourmap define a firstcolour look-up table and a second colour look-up table, respectively,such that the first and second colour look-up tables each comprise ndistinct colour values, where n equals the given number of encodedradiation intensity values.
 7. The method according to claim 6, whereinthe n distinct colour values of the second colourmap are eachindividually distinguishable for a human eye.
 8. The method according toclaim 1, wherein the electromagnetic radiation is thermal radiation. 9.The method according to claim 1, wherein the first colourmap comprisescolour values whose lightness and chromaticity values increase from afirst end of the first colourmap to a second end of the first colourmap.10. The method according to claim 1, wherein the first colourmap is asingle colour colourmap, and wherein the second colourmap is amulti-colour colourmap.
 11. The method according to claim 1, wherein themethod further comprises generating a binary fourth matrix of the samesize and shape as the first matrix, wherein the binary fourth matrixcomprises fourth matrix elements which are set to 1 for every non-zerovalue of the third matrix and the other fourth matrix elements are setto 0, or vice versa, and using the binary fourth matrix to combine thefirst colour image and the second colour image.
 12. The method accordingto claim 1, wherein the method further comprises adapting a displayluminance based on object background luminance.
 13. The method accordingto claim 12, wherein the method further comprises determining the objectbackground luminance and maintaining a predetermined ratio of thedisplay luminance to the object background luminance.
 14. The methodaccording to claim 12, wherein the method further comprises determiningthe object background luminance and varying a ratio of the displayluminance to the object background luminance between a lower thresholdand a higher threshold depending on an information value of the firstmatrix, where the lower threshold corresponds to a low informationvalue, while the higher threshold corresponds to a normal informationvalue.
 15. An imaging system for displaying an image on a see-throughdisplay, the imaging system comprising means for: obtaining a firstmatrix of electromagnetic radiation of an object, the first matrixcomprising first matrix elements representing radiation intensity valuesof corresponding locations of the object; dividing the first matrix intoa second matrix representing a first subset of the radiation intensityvalues below or equal to a predefined threshold, and a third matrixrepresenting a second subset of the radiation intensity values above thepredefined threshold; generating a first grayscale image with anenhanced contrast representing the first subset of the radiationintensity values from the second matrix; colouring the first grayscaleimage with a first colourmap to obtain a first colour image; generatinga second grayscale image representing the second subset of the radiationintensity values; colouring the second grayscale image with a secondcolourmap, which is different from the first colourmap, to obtain asecond colour image; combining the first colour image and the secondcolour image to obtain a combined colour image of the same size andshape as the first colour image or the second colour image; anddisplaying the combined colour image on the see-through display.
 16. Asystem for obtaining and displaying an image, comprising: a protectivehelmet configured to be worn by a user; a thermal sensor installed on afront part of the protective helmet and configured to capture a firstmatrix of radiation intensity values; a see-through display mounted tothe protective helmet and arranged in a field of view of the user; adata processing unit configured to: divide the first matrix into asecond matrix representing a first subset of the radiation intensityvalues below or equal to a predefined threshold, and a third matrixrepresenting a second subset of the radiation intensity values above thepredefined threshold, generate a first grayscale image with an enhancedcontrast representing the first subset of the radiation intensity valuesfrom the second matrix, colour the first grayscale image with a firstcolourmap to obtain a first colour image, generate a second grayscaleimage representing the second subset of the radiation intensity values,colour the second grayscale image with a second colourmap, which isdifferent from the first colourmap, to obtain a second colour image,combine the first colour image and the second colour image to obtain acombined colour image of the same size and shape as the first colourimage or the second colour image, and display the combined colour imageon the see-through display.